High level API of tensorflow¶
tensorflow選擇keras成為預設的high level api ?!¶
原本tf的high level api是tf.contrib¶
這時候我就要先來講一個小故事了
https://github.com/dboyliao/tf_snippets/blob/master/labs/batch_normalization.ipynb
https://github.com/dboyliao/tf_snippets/blob/master/labs/batch_normalization_v2.ipynb
In [1]:
from tensorflow.contrib.layers import batch_norm
| Dboy實作 | contrib... |
|---|---|
 |
 |
起手式¶
一定要記下tf跟keras目前寫起來對應的版號，雖然現在可能沒以前更新速度那麼嚴重，但這會是個省去以後很多麻煩的好習慣
In [2]:
import numpy as np
import keras
import tensorflow as tf
tf.__version__, keras.__version__
Out[2]:
Keras首先我們會從3個部分去建構出一個最簡單的nn¶
- layers : high level建構nn所需要的最小單元layer，選擇哪些layer來拼裝
- models : 把layer組裝起來的model
- optimizers : tune models的必需品
In [3]:
from keras.layers import Input, Dense, Activation
from keras.models import Model, Sequential
from keras.optimizers import SGD, RMSprop, Adam
In [4]:
from keras.utils import to_categorical
In [5]:
# mulilayer logistic
graph = tf.Graph()
with graph.as_default():
tf_images_batch = tf.placeholder(tf.float32, shape=[None, 28*28], name="images_batch")
tf_target_prob = tf.placeholder(tf.float32, shape=[None, 10], name="target_prob")
# Layer 1
tf_W1 = tf.Variable(0.05*np.random.randn(28*28, 200),
name="W1",
dtype=tf.float32)
tf_b1 = tf.Variable(np.zeros(200), dtype=tf.float32, name="b1")
tf_zscore1 = tf.nn.bias_add(tf.matmul(tf_images_batch, tf_W1),
tf_b1,
name="zcore_1")
tf_act1 = tf.nn.sigmoid(tf_zscore1, name="activation_1")
# Layer 2
tf_W2 = tf.Variable(0.05*np.random.randn(200, 10),
name="W2",
dtype=tf.float32)
tf_b2 = tf.Variable(np.zeros(10), dtype=tf.float32, name="b2")
tf_zscore2 = tf.nn.bias_add(tf.matmul(tf_act1, tf_W2), tf_b2, name="zscore_2")
tf_prob = tf.nn.softmax(tf_zscore2, 1)
# Cross-Entropy Loss
tf_loss = -tf.reduce_mean(tf_target_prob*tf.log(tf_prob))
# training
train_op = tf.train.GradientDescentOptimizer(1e-1).minimize(tf_loss)
saver = tf.train.Saver()
Keras組這台車¶
- Sequential
- Model
Sequential¶
In [6]:
mulilayer_logistic = [Dense(200, activation='sigmoid' ,name='Layer_1', input_shape=(28*28,)),
Dense(10, activation='softmax' ,name='Layer_2')]
model = Sequential(mulilayer_logistic)
model.summary()
Model¶
In [7]:
input_layer = Input(name='input0', shape=(28*28,), dtype='float32')
fc_layer_1 = Dense(200, activation='relu', name='Layer_1')(input_layer)
fc_layer_2 = Dense(10, activation='softmax', name='Layer_2')(fc_layer_1)
model = Model(inputs=[input_layer], outputs=[fc_layer_2])
model.summary()
In [8]:
model.compile(loss='categorical_crossentropy',
optimizer='sgd',
metrics=['accuracy'])
tensorflow讓車跑起來¶
In [12]:
batch_size=100
num_epochs = 50
log_every_epoch = 10
with tf.Session(graph=graph) as sess:
tf.global_variables_initializer().run()
for epoch in range(1, num_epochs+1):
current_epoch = fashion_mnist.train.epochs_completed
while fashion_mnist.train.epochs_completed <= current_epoch:
images_batch, labels_batch = fashion_mnist.train.next_batch(batch_size)
feed_dict = {tf_images_batch: images_batch,
tf_target_prob: labels_batch}
_, l = sess.run([train_op, tf_loss], feed_dict=feed_dict)
else:
if epoch % log_every_epoch == 0:
print("epoch: ", epoch)
print("current training loss:", l)
feed_dict = {tf_images_batch: fashion_mnist.test.images}
prob = sess.run(tf_prob, feed_dict=feed_dict)
acc = accuracy(np.argmax(prob, axis=1),
np.argmax(fashion_mnist.test.labels, axis=1))
print("testing acc: {:.2f}%".format(acc))
chkp_path = saver.save(sess,
global_step=epoch,
save_path="models_chkp/fashion_mnist_chkp-{:.4f}".format(acc))
Keras讓車跑起來¶
In [10]:
batch_size = 100
epochs = 5
model.fit(x_train, y_train,
batch_size=batch_size, epochs=epochs,
verbose=1, validation_data=(x_test, y_test))
Out[10]:
In [11]:
score = model.evaluate(x_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])
補個剛剛讓車跑之前keras也是需要準備資料的，不過這部分都用numpy處理，所以就不需要再拉tf出來比較，tf這邊也是包好的，我們再做多一點視覺化讓大家更了解data一些¶
In [55]:
from keras.datasets import mnist
In [56]:
(x_train, y_train), (x_test, y_test) = mnist.load_data()
In [22]:
from PIL import Image
import numpy as np
Image.fromarray(np.uint8(x_train[0].reshape(28, 28)))
Out[22]:
In [23]:
x_train.shape
Out[23]:
In [24]:
y_train.shape
Out[24]:
In [25]:
y_train[:10]
Out[25]:
把矩陣畫出來¶
In [15]:
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns
In [59]:
f, ax = plt.subplots(figsize=(13, 10))
sns.heatmap(x_train[0], annot=True, fmt="d", linewidths=.2, ax=ax)
Out[59]:
In [62]:
fig
Out[62]:
先轉成float32，把圖片從0-255壓到0-1之間¶
In [63]:
x_train = x_train.reshape(60000, 28*28)
x_test = x_test.reshape(10000, 28*28)
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')
x_train /= 255
x_test /= 255
print(x_train.shape[0], 'train samples')
print(x_test.shape[0], 'test samples')
把原本Y只有label變成one hot encoding用前面講過keras的utils¶
In [64]:
y_train = keras.utils.to_categorical(y_train)
y_test = keras.utils.to_categorical(y_test)
y_train[0]
Out[64]:
In [65]:
import fashion_mnist
(x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()
In [66]:
Image.fromarray(np.uint8(x_train[0].reshape(28, 28)))
Out[66]:
In [23]:
labels = {0:"T-shirt/top", 1:"Trouser", 2:"Pullover",
3:"Dress", 4:"Coat", 5:"Sandal", 6:"Shirt",
7:"Sneaker", 8:"Bag", 9:"Ankle boot"}
In [68]:
labels[y_train[0]]
Out[68]:
一樣我們來看看，矩陣畫出來長的樣子¶
In [36]:
f, ax = plt.subplots(figsize=(15, 10))
sns.heatmap(x_train[0], annot=True, fmt="d", linewidths=.2, ax=ax)
Out[36]:
In [70]:
fig
Out[70]:
我們回頭來看前面用keras快速構建出來train出來的model的結果¶
In [12]:
#model.predict_classes
model.predict(x_train[:1])
Out[12]:
In [17]:
pd.DataFrame(model.predict(x_train[:1])).transpose().plot(kind='bar')
Out[17]:
In [18]:
np.argmax(model.predict(x_test[:20]), axis=1)
Out[18]:
In [19]:
np.argmax(y_test[:20], axis=1)
Out[19]:
In [20]:
error_arr = np.argmax(y_test[:100], axis=1) != np.argmax(model.predict(x_test[:100]), axis=1)
error_arr[:20]
Out[20]:
In [21]:
[i for i, bo in enumerate(error_arr) if bo==True][:10]
Out[21]:
In [24]:
pd.DataFrame(model.predict(x_test[12:13]).reshape(-1,), index=list(labels.values())).plot(kind='bar')
Out[24]:
In [27]:
display(labels[np.argmax(y_test[12:13])])
display(Image.fromarray(np.uint8(x_test[12].reshape(28, 28)*255)))
In [31]:
fig
Out[31]:
In [29]:
pd.DataFrame(model.predict(x_test[20:21]).reshape(-1,), index=list(labels.values())).plot(kind='bar')
Out[29]:
In [30]:
display(labels[np.argmax(y_test[20:21])])
display(Image.fromarray(np.uint8(x_test[20].reshape(28, 28)*255)))
In [33]:
fig
Out[33]:
伸手到keras內部探索¶
In [38]:
model.compile(loss='categorical_crossentropy',
optimizer='sgd',
metrics=['accuracy'])
先拿用文字寫的loss function開刀¶
寫深度學習最討厭的就是這種用string定出來的東西...¶
前面我們看過兩種model大概可以感覺或猜出來Sequential是Model包起來的，所以Sequential雖然方便但是如果要一些比較特別的架構就需要用Model，但這邊不是我們的重點，我們要的是看看model.compile做了哪些事¶
keras.models.py¶
from .engine.training import Model
keras.engine.training.py¶
from .. import losses
class Model(Container):
def compile(self, optimizer, loss, metrics=None, loss_weights=None,
sample_weight_mode=None, weighted_metrics=None,
target_tensors=None, **kwargs):
loss = loss or {}
self.optimizer = optimizers.get(optimizer)
self.sample_weight_mode = sample_weight_mode
self.loss = loss
self.loss_weights = loss_weights
# Prepare loss functions.
if isinstance(loss, dict):
for name in loss:
if name not in self.output_names:
pass
loss_functions = []
for name in self.output_names:
if name not in loss:
pass
loss_functions.append(losses.get(loss.get(name)))
elif isinstance(loss, list):
if len(loss) != len(self.outputs):
pass
loss_functions = [losses.get(l) for l in loss]
else:
loss_function = losses.get(loss)
loss_functions = [loss_function for _ in range(len(self.outputs))]
self.loss_functions = loss_functions
In [8]:
keras.losses.get??
def get(identifier):
if identifier is None:
return None
if isinstance(identifier, six.string_types):
identifier = str(identifier)
return deserialize(identifier)
elif callable(identifier):
return identifier
else:
raise ValueError('Could not interpret '
'loss function identifier:', identifier)
In [39]:
keras.losses.get('categorical_crossentropy')
Out[39]:
In [10]:
keras.losses.categorical_crossentropy??
def categorical_crossentropy(y_true, y_pred):
return K.categorical_crossentropy(y_true, y_pred)
In [40]:
keras.losses.get('mse')
Out[40]:
In [12]:
keras.losses.mean_squared_error??
def mean_squared_error(y_true, y_pred):
return K.mean(K.square(y_pred - y_true), axis=-1)
keras.losses.py¶
from . import backend as K
keras.backend.__init__.py¶
# Set backend based on KERAS_BACKEND flag, if applicable.
if 'KERAS_BACKEND' in os.environ:
_backend = os.environ['KERAS_BACKEND']
assert _backend in {'theano', 'tensorflow', 'cntk'}
_BACKEND = _backend
# Import backend functions.
if _BACKEND == 'cntk':
sys.stderr.write('Using CNTK backend\n')
from .cntk_backend import *
elif _BACKEND == 'theano':
sys.stderr.write('Using Theano backend.\n')
from .theano_backend import *
elif _BACKEND == 'tensorflow':
sys.stderr.write('Using TensorFlow backend.\n')
from .tensorflow_backend import *
else:
raise ValueError('Unknown backend: ' + str(_BACKEND))
In [41]:
dir(keras.backend.tensorflow_backend)
Out[41]:
def mean_squared_error(y_true, y_pred):
return K.mean(K.square(y_pred - y_true), axis=-1)
def categorical_crossentropy(y_true, y_pred):
return K.categorical_crossentropy(y_true, y_pred)
cross-entropy¶
$$ H_{\hat{y}}(y) = -\sum_i \hat{y}_i \log(y_i) $$In [42]:
def custom_crossentropy(y_true, y_pred):
return -tf.reduce_sum(y_true * keras.backend.log(y_pred), reduction_indices=len(y_pred.get_shape())-1)
In [43]:
def custom_crossentropy(y_true, y_pred):
return -keras.backend.sum(y_true * keras.backend.log(y_pred), axis=len(y_pred.get_shape())-1)
In [44]:
def custom_crossentropy(y_true, y_pred):
y_pred /= tf.reduce_sum(y_pred, axis=len(y_pred.get_shape())-1, keep_dims=True)
_epsilon = keras.backend.tensorflow_backend._to_tensor(keras.backend.epsilon(), y_pred.dtype.base_dtype)
y_pred = tf.clip_by_value(y_pred, _epsilon, 1. - _epsilon)
return -tf.reduce_sum(y_true * keras.backend.log(y_pred), reduction_indices=len(y_pred.get_shape())-1)
In [45]:
def custom_crossentropy(y_true, y_pred):
y_pred /= keras.backend.sum(y_pred, axis=len(y_pred.get_shape())-1, keepdims=True)
_epsilon = keras.backend.epsilon()
y_pred = keras.backend.clip(y_pred, _epsilon, 1. - _epsilon)
return -keras.backend.sum(y_true * keras.backend.log(y_pred), axis=len(y_pred.get_shape())-1)
In [47]:
layers = [Dense(256, activation='relu' ,name='fc0', input_shape=(28*28,)),
Dropout(0.5, name='dropout0'),
Dense(128, activation='relu' ,name='fc1'),
Dropout(0.5, name='dropout2'),
Dense(10, activation='softmax', name='predict_layer'),]
model = Sequential(layers)
model.summary()
optimizers順便換一下這邊剛剛最前面已經import過，這次讓我們也來試試¶
In [48]:
from keras.optimizers import SGD, Adam, RMSprop
adam = Adam(lr=0.001, beta_1=0.9, beta_2=0.99)
In [49]:
model.compile(loss=custom_crossentropy,
optimizer=adam,
metrics=['accuracy'])
In [33]:
epochs = 10
batch_size = 256
history = model.fit(x_train, y_train, batch_size=batch_size,
epochs=epochs, verbose=1,
validation_data=(x_test, y_test))
User Keras to play Autoencoder¶
Autoendoer¶
$$ F(x) \simeq x \\ F(x) = Decoder(Encoder(x)) $$
好的開始建我們的autoencoder¶
- 這邊注意到我先把weights的init提取出來，先不用管他，這是keras dense layer預設的init
In [52]:
act = 'sigmoid'
layer_init = 'glorot_uniform'
encoding_dim = 2
input_ = Input(name='input_0', shape=(28*28,))
encoded = Dense(128, activation=act, kernel_initializer=layer_init, name='encode_1')(input_)
encoded = Dense(64, activation=act, kernel_initializer=layer_init, name='encode_2')(encoded)
encoded = Dense(10, activation=act, kernel_initializer=layer_init, name='encode_3')(encoded)
encoded = Dense(encoding_dim, activation='sigmoid', kernel_initializer=layer_init, name='encoder')(encoded)
decoded = Dense(10, activation=act, kernel_initializer=layer_init, name='decode_1')(encoded)
decoded = Dense(64, activation=act, kernel_initializer=layer_init, name='decode_2')(decoded)
decoded = Dense(128, activation=act, kernel_initializer=layer_init, name='decode_3')(decoded)
decoded = Dense(784, activation='sigmoid', kernel_initializer=layer_init, name='decode_4')(decoded)
autoencoder = Model(inputs=[input_], outputs=[decoded])
In [53]:
autoencoder.summary()
In [54]:
def tf_sigmoid_cross_entropy(target, output):
_epsilon = keras.backend.tensorflow_backend._to_tensor(keras.backend.epsilon(), output.dtype.base_dtype)
output = tf.clip_by_value(output, _epsilon, 1 - _epsilon)
output = tf.log(output / (1 - output))
return tf.nn.sigmoid_cross_entropy_with_logits(labels=target, logits=output)
def check_folder(log_path, del_mode):
if del_mode:
if os.path.exists(log_path):
for f in os.listdir(log_path):
os.remove(os.path.join(log_path, f))
return log_path
else:
return log_path+'0'
這邊有個tensorboard callback基本上就是keras結合tensorboard的用法，在不動到keras底下，用法就這樣，然後tensorboard開法跟前面講得一樣加log_dir就可以¶
In [55]:
tensorboard_callback = keras.callbacks.TensorBoard(log_dir=check_folder('logs/glorot_uniform_init', 1),
histogram_freq=1,
write_graph=True, write_grads=True)
autoencoder.compile(loss=tf_sigmoid_cross_entropy,
optimizer=Adam(lr=0.001))
這邊我們把encoder拿出來，因為encoder最後output是一個二維向量我們可以把它畫在x,y平面上, 這就是autoencoder可以拿到的feature vector¶
In [58]:
encoder = Model(inputs=[input_], outputs=[encoded])
encoder.summary()
In [59]:
encoder.predict(x_test[:5])
Out[59]:
先來看一下畫出來的結果，仔細看一下座標¶
In [61]:
f
Out[61]:
調整到sigmoid的區間0,1再看一次...¶
In [63]:
fig
Out[63]:
沒關係我們還是train看看，但是因為值有可能會爆掉，所以先把tensorboard注解掉，tensorboard callback帶進fit底下就可以¶
In [64]:
autoencoder.fit(x_train, x_train,
epochs=7, batch_size=128,
shuffle=True, verbose=1,
validation_data=(x_test, x_test),
#callbacks=[tensorboard_callback]
)
Out[64]:
看一下可怕的結果¶
In [66]:
fig
Out[66]:
glorot init¶
http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf
這邊有一篇詳細的論完講了keras這個dense layer預設的init，這邊我們就不詳細解釋，為什麼很多人在一般的classfication或是用relu當activate的時候效果都還不錯，甚至有些人會說keras隨便預設都train的比自幹tf的好？當然不是用keras的人強...是keras聚集了很多的研究經驗tune好很多東西了，只能說Chollet大神很猛，基本去讀keras source code就會看到什麼funtion是用哪個論文實作的註解
在玩起autoencoder前我們來試試keras中的initializers¶
In [67]:
from keras.initializers import glorot_uniform, glorot_normal, RandomNormal, RandomUniform, TruncatedNormal
先來測TruncatedNormal至於為什麼？...因為我很久以前實作autoencoder當時是用TruncatedNormal，在我直接用keras預設init無法時就想到，相比前面穩定不少¶
In [77]:
layer_init = TruncatedNormal(mean=.0, stddev=1.)
act = 'sigmoid'
encoding_dim = 2
input_ = Input(name='input_0', shape=(28*28,))
encoded = Dense(128, activation=act, kernel_initializer=layer_init, name='encode_1')(input_)
encoded = Dense(64, activation=act, kernel_initializer=layer_init, name='encode_2')(encoded)
encoded = Dense(10, activation=act, kernel_initializer=layer_init, name='encode_3')(encoded)
encoded = Dense(encoding_dim, activation='sigmoid', kernel_initializer=layer_init, name='encoder')(encoded)
decoded = Dense(10, activation=act, kernel_initializer=layer_init, name='decode_1')(encoded)
decoded = Dense(64, activation=act, kernel_initializer=layer_init, name='decode_2')(decoded)
decoded = Dense(128, activation=act, kernel_initializer=layer_init, name='decode_3')(decoded)
decoded = Dense(784, activation='sigmoid', kernel_initializer=layer_init, name='decode_4')(decoded)
autoencoder = Model(inputs=[input_], outputs=[decoded])
In [78]:
tensorboard_callback = keras.callbacks.TensorBoard(log_dir=check_folder('logs/truncated_normal_init', 1),
histogram_freq=1,
write_graph=True, write_grads=True)
autoencoder.compile(loss=tf_sigmoid_cross_entropy,
optimizer=Adam(lr=0.001))
先來看看init完的feature vertor¶
In [80]:
fig
Out[80]:
train看看，加callback基本沒爆就是好的...¶
In [82]:
autoencoder.fit(x_train, x_train,
epochs=6, batch_size=128,
shuffle=True, verbose=1,
validation_data=(x_test, x_test),
#callbacks=[tensorboard_callback]
)
Out[82]:
In [12]:
fig
該不會是uniform跟normal的關係吧？直覺想起來應該也不太是¶
In [13]:
layer_init = glorot_normal()
act = 'sigmoid'
encoding_dim = 2
input_ = Input(name='input_0', shape=(28*28,))
encoded = Dense(128, activation=act, kernel_initializer=layer_init, name='encode_1')(input_)
encoded = Dense(64, activation=act, kernel_initializer=layer_init, name='encode_2')(encoded)
encoded = Dense(10, activation=act, kernel_initializer=layer_init, name='encode_3')(encoded)
encoded = Dense(encoding_dim, activation='sigmoid', kernel_initializer=layer_init, name='encoder')(encoded)
decoded = Dense(10, activation=act, kernel_initializer=layer_init, name='decode_1')(encoded)
decoded = Dense(64, activation=act, kernel_initializer=layer_init, name='decode_2')(decoded)
decoded = Dense(128, activation=act, kernel_initializer=layer_init, name='decode_3')(decoded)
decoded = Dense(784, activation='sigmoid', kernel_initializer=layer_init, name='decode_4')(decoded)
autoencoder = Model(inputs=[input_], outputs=[decoded])
autoencoder.summary()
看起來不是...直接不train了¶
In [14]:
fig
試試Random Unifrom好了印證一下¶
In [15]:
layer_init = RandomUniform(minval=-1, maxval=1)
act = 'sigmoid'
encoding_dim = 2
input_ = Input(name='input_0', shape=(28*28,))
encoded = Dense(128, activation=act, kernel_initializer=layer_init, name='encode_1')(input_)
encoded = Dense(64, activation=act, kernel_initializer=layer_init, name='encode_2')(encoded)
encoded = Dense(10, activation=act, kernel_initializer=layer_init, name='encode_3')(encoded)
encoded = Dense(encoding_dim, activation='sigmoid', kernel_initializer=layer_init, name='encoder')(encoded)
decoded = Dense(10, activation=act, kernel_initializer=layer_init, name='decode_1')(encoded)
decoded = Dense(64, activation=act, kernel_initializer=layer_init, name='decode_2')(decoded)
decoded = Dense(128, activation=act, kernel_initializer=layer_init, name='decode_3')(decoded)
decoded = Dense(784, activation='sigmoid', kernel_initializer=layer_init, name='decode_4')(decoded)
autoencoder = Model(inputs=[input_], outputs=[decoded])
autoencoder.summary()
In [17]:
autoencoder.compile(loss=tf_sigmoid_cross_entropy,
optimizer=Adam(lr=0.001))
應該就是glorot的問題了...¶
In [18]:
fig
In [19]:
autoencoder.fit(x_train, x_train,
epochs=6, batch_size=128,
shuffle=True, verbose=1,
validation_data=(x_test, x_test),
#callbacks=[tensorboard_callback]
)
Out[19]:
整個群有分開一點了¶
In [20]:
fig
In [33]:
layer_init = RandomNormal(mean=.0, stddev=1.)
act = 'sigmoid'
encoding_dim = 2
input_ = Input(name='input_0', shape=(28*28,))
encoded = Dense(128, activation=act, kernel_initializer=layer_init, name='encode_1')(input_)
encoded = Dense(64, activation=act, kernel_initializer=layer_init, name='encode_2')(encoded)
encoded = Dense(10, activation=act, kernel_initializer=layer_init, name='encode_3')(encoded)
encoded = Dense(encoding_dim, activation='sigmoid', kernel_initializer=layer_init, name='encoder')(encoded)
decoded = Dense(10, activation=act, kernel_initializer=layer_init, name='decode_1')(encoded)
decoded = Dense(64, activation=act, kernel_initializer=layer_init, name='decode_2')(decoded)
decoded = Dense(128, activation=act, kernel_initializer=layer_init, name='decode_3')(decoded)
decoded = Dense(784, activation='sigmoid', kernel_initializer=layer_init, name='decode_4')(decoded)
autoencoder = Model(inputs=[input_], outputs=[decoded])
autoencoder.summary()
In [34]:
autoencoder.compile(loss=tf_sigmoid_cross_entropy,
optimizer=Adam(lr=0.001))
In [36]:
fig
In [37]:
tensorboard_callback = keras.callbacks.TensorBoard(log_dir=check_folder('logs/random_normal_init', 1),
histogram_freq=1,
write_graph=True, write_grads=True)
autoencoder.fit(x_train, x_train,
epochs=6, batch_size=128,
shuffle=True, verbose=1,
validation_data=(x_test, x_test),
callbacks=[tensorboard_callback]
)
Out[37]:
In [38]:
fig
In [40]:
fig
來看看Decode回來的結果¶
In [45]:
fig
回頭看一下爛的init完的結果¶
- 大家也可以試一下好的init完的decode結果
In [ ]:
fig, axes = plt.subplots(4, 9, figsize=(16, 8))
decoded_imgs = [Image.fromarray(np.uint8(arr.reshape(28, 28)*255)) for arr in autoencoder.predict(x_test[:18])]
ori_imgs = [Image.fromarray(np.uint8(x_test[i].reshape(28, 28)*255)) for i in range(18)]
pil_images = decoded_imgs + ori_imgs
for i, ax in enumerate(axes.ravel()):
im = ax.imshow(np.asarray(pil_images[i]), cmap='gray', interpolation='lanczos')
if i > 17:
i -= 18
ax.set_title(labels[y_test[i]] + ' %d'%y_test[i])
ax.set_xticks(np.arange(0, 28, 28))
ax.set_yticks(np.arange(0, 28, 28))
ax.grid(color='k', linestyle='-', linewidth=0)
In [49]:
fig
bad init result¶
動手玩時間¶
- 不同的init特性？
- 如果把activation function換掉呢？
- mse跟cross-entropy會影響嗎？
- optimizer也換換看？RMSprop好像在做這種類型的事比較常用？Adam比較沒法收斂到最好的結果？
- deocder前的feature vector自己生？會decode出什麼樣子？
- 換layer玩玩？？！！試試看用Convolutional layer玩？？
- autoencoder還可以拿來幹嘛？denosie?
- 在進階的來玩VAE？？？！！！
- VAE+CONV？？？好了沒時間了
Play Transfer Learning¶
- 什麼是transfer learning?
- 為什麼需要transfer learning?
- transfer learning的架構要長什麼樣子？
我們一樣先用fashion mnist已經很熟悉的例子來試試¶
In [87]:
(x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()
labels = {0:"T-shirt/top", 1:"Trouser", 2:"Pullover",
3:"Dress", 4:"Coat", 5:"Sandal", 6:"Shirt",
7:"Sneaker", 8:"Bag", 9:"Ankle boot"}
我們記得剛剛玩autoencoder的時候已經知道，有些東西是蠻像的，或者說存在的差異可能有點像？¶
- 沒關係我們先簡單的把資料切開，用0-4, 5-9來區分，剛剛記得2,6很難分算是比較像的，所以應該有一些相同特徵
- 我們用前面五個去學出feature layer用後面五個做測試是否能把學到的特徵轉移做出新的分類器？！
複習一下numpy操作¶
In [88]:
y_train < 5
Out[88]:
拿出我們要的0-4的當作training data¶
In [89]:
x_train_f5 = x_train[y_train < 5]
y_train_f5 = y_train[y_train < 5]
x_test_f5 = x_test[y_test < 5]
y_test_f5 = y_test[y_test < 5]
ok開始建構我們的transfer learning nn¶
In [92]:
feature_layers = [
Dense(512, activation='relu' ,name='dense0_0', input_shape=(784,)),
Dense(384, activation='relu' ,name='dense0_1'),
Dense(256, activation='relu' ,name='dense0_2'),
Dense(128, activation='relu' ,name='dense0_3'),
]
retrain_layer = [
Dense(56, activation='relu' ,name='dense1_0'),
Dropout(0.5, name='dropout0_1'),
Dense(num_classes, activation='softmax', name='predict_layer'),
]
model = Sequential(feature_layers + retrain_layer)
In [93]:
model.summary()
前五類的pre train¶
In [95]:
%%time
model.fit(x_train_f5, y_train_f5,
batch_size=batch_size, epochs=epochs,
verbose=1,
validation_data=(x_test_f5, y_test_f5))
score = model.evaluate(x_test_f5, y_test_f5, verbose=0)
print('Test accuracy:', score[1])
我們把feature layer固定起來不能再被train，只有負責分類的layer可以繼續train¶
In [96]:
feature_layers[0].trainable
Out[96]:
In [97]:
for layer in feature_layers:
layer.trainable = False
In [98]:
model.summary()
再來就用我們pre train好的nn再來做訓練¶
- 先準備data把5-9切出來
In [99]:
x_train_last5 = x_train[y_train >= 5]
y_train_last5 = y_train[y_train >= 5] - 5
x_test_last5 = x_test[y_test >= 5]
y_test_last5 = y_test[y_test >= 5] - 5
我們可以調整transfer learning的數量，假設這些data沒有那麼足夠的label data¶
In [100]:
less_data_num = 5000
In [85]:
x_train_last5 = x_train_last5[:less_data_num]
y_train_last5 = y_train_last5[:less_data_num]
In [86]:
model.compile(loss=categorical_crossentropy,
optimizer=Adam(),
metrics=['accuracy'])
我們先來看一下在還沒transfer learning前這個model直接predict的狀況？¶
In [121]:
score = model.evaluate(x_test_last5, y_test_last5, verbose=0)
print('Test accuracy:', score[1])
居然一開始就有19% ??!! 好奇看一下哪些被猜對了¶
In [165]:
correct_arr = np.argmax(y_test_last5[:], axis=1) == np.argmax(model.predict(x_test_last5[:]), axis=1)
correct_ind = [i for i, bo in enumerate(correct_arr) if bo==True]
correct_ind[:10]
Out[165]:
In [168]:
fig
Out[168]:
最後跑了50個epoch後準確率只有到80比原本的差了10%而且也許用同樣的data直接train能更好？¶
- features沒有被前面的layer有效提取出來？
- 前面的類沒有足夠的特徵作為後面的類可切開？
- data數量太少？
In [87]:
%%time
epochs = 50
model.fit(x_train_last5, y_train_last5,
batch_size=batch_size, epochs=epochs,
verbose=1,
validation_data=(x_test_last5, y_test_last5))
score = model.evaluate(x_test_last5, y_test_last5, verbose=0)
print('Test accuracy:', score[1])
選一個更好的架構來做feature engineering¶
- Convolution, Maxpooling 這邊我們開始讓大家有點認識體會但不會細講，之後課程會更深入的探討conv做的事情
In [88]:
from keras.layers import Conv2D, MaxPool2D
In [91]:
feature_layers = [
Conv2D(filters, kernel_size, padding='valid',
input_shape=input_shape, name='conv1_1'),
Activation('relu'),
Conv2D(filters, kernel_size, padding='valid', name='conv1_2'),
Activation('relu'),
MaxPool2D(pool_size=pool_size, name='maxpool1'),
Dropout(0.25, name='dropout_0'),
Conv2D(filters, kernel_size, padding='valid',
input_shape=input_shape, name='conv2_1'),
Activation('relu'),
Conv2D(filters, kernel_size, padding='valid', name='conv2_2'),
Activation('relu'),
MaxPool2D(pool_size=pool_size, name='maxpool2'),
Dropout(0.25, name='dropout_1'),
Flatten(),
]
retrain_layer = [
Dense(56, activation='relu' ,name='dense1'),
Dropout(0.5, name='dropout_2'),
Dense(num_classes, activation='softmax', name='predict_layer'),
]
In [92]:
model = Sequential(feature_layers + retrain_layer)
model.summary()
我們train起來每個epoch大約30秒準確度也算相當的提升，那問題是這個架構能不能成功找到前後資料中可以被重新組合做分類的feature呢？¶
In [98]:
%%time
model.fit(x_train_f5, y_train_f5,
batch_size=batch_size, epochs=epochs,
verbose=1,
validation_data=(x_test_f5, y_test_f5))
score = model.evaluate(x_test_f5, y_test_f5, verbose=0)
print('Test accuracy:', score[1])
我們一樣需要把feature layers的參數定住¶
In [100]:
for layer in feature_layers:
layer.trainable = False
model.summary()
還有一半的參數可以train¶
In [101]:
x_train_last5 = x_train[y_train >= 5]
y_train_last5 = y_train[y_train >= 5] - 5
x_test_last5 = x_test[y_test >= 5]
y_test_last5 = y_test[y_test >= 5] - 5
這次我們直接用更少量的data來做transfer¶
In [102]:
less_data_num = 1500
In [103]:
x_train_last5 = x_train_last5.reshape((-1,)+ input_shape)[:less_data_num]
x_test_last5 = x_test_last5.reshape((-1,)+ input_shape)
x_train_last5 = x_train_last5.astype('float32')
x_test_last5 = x_test_last5.astype('float32')
x_train_last5 /= 255
x_test_last5 /= 255
print(x_train_last5.shape[0], 'train samples')
print(x_test_last5.shape[0], 'test samples')
y_train_last5 = keras.utils.to_categorical(y_train_last5, num_classes)[:less_data_num]
y_test_last5 = keras.utils.to_categorical(y_test_last5, num_classes)
因為每次epoch數量變少，所以相對的速度變很快，20個epoch後還算不錯，我們可以再測試再多跑幾個epoch能不能更好？¶
In [106]:
%%time
epochs = 20
model.fit(x_train_last5, y_train_last5,
batch_size=batch_size, epochs=epochs,
verbose=1,
validation_data=(x_test_last5, y_test_last5))
score = model.evaluate(x_test_last5, y_test_last5, verbose=0)
print('Test accuracy:', score[1])
再跑50個epoch後成功得到94%準確率，比原本的transfer之前更好？！¶
In [107]:
epochs = 50
model.fit(x_train_last5, y_train_last5,
batch_size=batch_size, epochs=epochs,
verbose=1,
validation_data=(x_test_last5, y_test_last5))
score = model.evaluate(x_test_last5, y_test_last5, verbose=0)
print('Test accuracy:', score[1])
動手玩時間¶
- 有沒有辦法讓前面fc變成有用的transfer？一次固定一層train?後面retrain層數增加？feature層不要變小？......
- 用VGG來玩transfer？？？！！！ https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html